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ABSTRACT 

Three-dimensional perception will be essential to the 
development of versatile robotics systems capable of 
handling complicated manufacturing tasks in the 
factory of the future and in providing the high 
accuracy measurements needed in flexible 

manufacturing and quality control. The program 

described here will develop the potential of moire 
techniques to provide this greatly needed capability in 
vision systems and automated measurements, and 
demonstrate Artificial Intelligence (AI) techniques to 
take advantage of the unique strengths of moire 
sensing. Moire techniques provide a means of 
optically manipulating the complex visual data in a 
three-dimensional scene into a form which can be 
easily and quickly analyzed by computers. This type 
of optical data manipulation will provide higher 
productivity through integrated automation, 
producing a higher quality product while reducing 
computer and mechanical manipulation requirements 
and thereby the cost and time of production. The 
objective of this program will be to develop this 
nondestructive evaluation technique such as to be 
capable of full-field range measurement and 3- 
dimensional scene analysis. 


BACKGROUND 

One of the most powerful senses available to humans 
is vision. Vision allows us to collect and analyze vast 
amounts of empirical data at an astounding rate. 
Many years of research have been devoted to 
developing sophisticated image processing systems for 
medical and military applications. Modern computing 
capabilities have greatly aided these systems by 
handling the vast amounts of information at faster 
and faster speeds. These systems often use large 
computing systems, yet still often require highly 
skilled operators. For a vision system to be practical 
for industrial application it must be versatile, fast 
(less than 0.1 seconds typically), inexpensive, and it 
must require a minimal amount of human operator 
support [1,2], Given these requirements and the 
current limits of small computer systems, the amount 
of data which can be processed is limited. 


There are currently three basic approaches to three- 
dimensional machine vision; range finding including 
structured lighting, stereo or binocular vision, and 
gray scale methods[3-16]. There are varying degrees 
to the range finding approach. A simple version is to 
focus a beam of light on the object at a given 
distance. As the object surface moves closer or more 
distant, the spot on the object surface will enlarge, 
the size of the spot being directly proportional to the 
change in surface height. The most popular versions 
use the triangulation method where a beam of light is 
projected onto the object’s surface at some angle and 
the image of this spot or line of light is viewed at 
some other angle. As the object distance changes a 
spot of light on the surface will move along the 
surface by (change in spot position) = (change in 
distance) x (tan(incident angle) + tan(viewing angle)). 
If a line is projected onto the surface by imaging or 
by scanning a beam, the line will deform as it moves 
across a contoured surface as each point of the line 
move as described above[5-9]. Other range -finding 
systems often use selected point measurements to 
supplement some other system [10], Other systems use 
multiple lines or patterns such as reticles to cover 
more area at a time[4,ll,12]. 

Stereo or binocular vision methods work on the same 
principle as human vision by obtaining parallax 
information by viewing the object from two different 
perspectives (as our two eyes do) [4, 13- 15], The two 
views must be matched up and then the difference in 
the position of a correlated feature gives the depth 
information from the geometry of the two view points 
(similar to the triangulation methods). These 
methods are full-field, thereby keeping the data 
acquisition time to a minimum, and they do provide 
all the 2-D feature information (in fact they depend 
on these surface features). There has been some 
success using special purpose correlation hardware in 
a method similar to those developed for military 
surveillance industrial. Generally the software 
manipulation is necessarily intensive[4,15]. 

Shape form shading methods work on the principle 
that a uniform field of light incident on a uniformly 
reflecting surface will vary in intensity depending on 
its angle of incidence. That is, as the angle of 
incidence increases, a given portion of the light field 


‘' •^CEDING PAGE BLANK NOT FILM ID 


413 



will be spread over a wider area of the surface, 
thereby decreasing how brightly that area is 
illuminated[4,16]. This system does not provide an 
absolute measurement relative to the machine 
coordinates (the other two approaches do provide this 
information) since it is insensitive to step changes. 
An auxiliary system such as a rangefinder would need 
to be added. 

Precision full-field part measurement is an even more 
complicated task. Manual point-by-point 

measurements are often long and laborious. Without 
full-field object data, some imperfections can be 
missed altogether. There are some coordinate 
measuring machines on the market which can be 
programmed to measure one or a few specific 
parts[l7]. This approach saves human labor and 
human error but does not necessarily work any faster. 
Therefore, measurements for quality control are often 
limited to a spot check system. A versatile automated 
contouring system capable of measuring either large 
or even small areas at a time would provide the 
opportunity for better and more complete inspections. 
For many applications, speed is a very important 
factor as well. An approach which simplifies the 
required computing capabilities while providing added 
data processing speed is to use some form of optical 
preprocessing of the visual image. Optical data 
manipulation and encoding acts on an entire visual 
scene simultaneously in a parallel fashion. Parallel 
processing of this type is very fast and can be used to 
arrange or sort information into a form which can 
more easily and more quickly be analyzed with a 
computer than by direct digitizing techniques. 
Unwanted information can be disposed of and the 
desired information can be highlighted, sorted, or 
encoded for easier manipulation. In particular, moire 
techniques can transform the out-of-plane shape 
information of an object into a two-dimensional 
intensity pattern in such a way as to separate the 
third dimension completely from the other two 
without losing information. 

TECHNICAL DISCUSSION 

Two main technologies support this program: moire 
interferometry for optical processing and data 
acquisition, and iterative model-driven constraint 
directed reasoning for scene analysis. 

Moire interferometry is a full-field, noncontact 
measurement technique[l8,19]. A moire pattern is 
made by forming a subject grating, by projecting, 
shadowing, or contacting a grating onto the object to 
be measured, and comparing this grating to some 
reference grating by overlaying the two grating 
images. If the reference grating is a straight line 
grating, the beat pattern between the two gratings 
will form a contour map of the object’s surface in the 
same way that a topographic map delineates the 
contours of the land. A diagram of a simple 
projection moire system is shown in Figure 1. In this 
case, the grating is imaged onto the surface, then the 
surface is imaged back to a reference plane, from an 
angle different from the illumination angle, and the 
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Figure 1. Diagram of moire setup. 

resulting image is viewed through the reference 
grating. As an illustration of the pattern produced, 
Figure 2 shows a contour moire of a turbine blade. 
For this example a grating was placed in front of the 
statue and simply shadowed onto the statue, then the 
shadow was viewed through the original grating. This 
shadow moire effect can often be seen in everyday 
situations such as the patterns seen when an object is 
viewed through a screen door or through sheer woven 
draperies. 



Figure 2. Moire pattern on a turbine blade. 


If the reference grating is made by recording the 
image of the object grating, then the moire pattern 
can be used to show only differences between the 
reference object or reference state of an object and 
some new object or state. In this way, moire can be 
used to show only deviations from a good part. When 
applied to an on-line inspection system this difference 
moire approach greatly reduces the amount of 
information to be analyzed to determine if a part is 
within tolerances or to simply identify the part. 

The sensitivity of moire contouring is given by the 
same relationship as other triangulation methods, ie. 
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Z = p/(tan(i) + tan(v)) 
where p = the grating period on the 
object 

i = the angle of incidence of 
the projected pattern 

v == the viewing angle 

Z = the sensitivity per fringe 

(the projected beam has been assumed to be 
approximately collimated, though this is not necessary 
in practice) 

Typically, with digitizing methods a fringe can be 
measured to one tenth of a fringe. With such 
methods a sensitivity of 0.001 inches over a full field 
of a few square feet is not unreasonable with higher 
sensitivity over small areas. For example, over a one 
square foot area, the sensitivity could exceed 0.0005 
inches, and over one square inch changes as small as 
10 microinches have been recorded using moire 
techniques[18]. The particular sensitivity that would 
be practical would depend on many factors such as 
part geometry, system resolution, etc.. The sensitivity 
of the moire pattern can be tailored to fit the 
requirement by simply adjusting the grating period. 

Since moire is full-field, the contour of an entire area 
of an object can be mapped out at one time and 
recorded during a single video frame. This allows 
anomalies as well as large-scale shape features to be 
viewed and measured to the same precision and at the 
same point in time (important in situations where 
thermal drifts or other factors may be warping the 
part with time). The instantaneous contour plot can 
be viewed Immediately with high-accumey numerical 
results available after a reasonable amount of analysis 
time. Such real-time viewing of a component is a 
common practice in fabricating high-quality optical 
components to tolerances of a few microinches. Many 
facilities and interferometer manufacturers have 
incorporated computer systems to digitize and analyze 
fringe data much like those from moire 
con touring[l8, 20-23]. As an example of such 

digitizing methods, Figure 3 shows a high sensitivity 
moire pattern (about 0.04 inches per fringe) of a 
machined part with bevelled surfaces. Figure 4 shows 
the computer generated isometric plot of the surface 
shape. The actual value of each point on the disk 
was then available to the computer to the spacial 
resolution of the computer model. In this case the 
depth information was available to about 0.001 
inches. 

As with the line projection techniques, the depth 
information is effectively encoded by the moire 
interferometer into a 2-dimensional map that is both 
easy for the computer system to record, since it is 
only from one perspective, and is independent of 2-D 
features on the surface of the object. This ability 
makes the 2-D features separable from the depth 
information which means the 2-D outlines can be 
analyzed separately using well established vision 
algorithms. 



Figure 3. Moire patern on a machined part. 



Figure 4. Shape information from moire. 

Optically specular surfaces can and are routinely 
inspected using moire methods, though the method 
used for contouring specular surfaces is typically 
different than that employed for diffuse surfaces. It is 
typically easier to treat diffuse surface and specular 
surface contouring differently, but the results and the 
analysis are largely the same. 

Peaks and valleys can easily be distinguished and 
absolute measurements made by a variety of methods. 
The sign of the slope being measured can be 
determined by a wide range of methods such as 
moving one grating with respect to the other (the 
fringes will move in different directions depending on 
whether the slope is positive or negative) or by 
moving the object between two consecutive recordings 
or by using mismatched gratings. These methods are 
very familiar to people working in moire 
interferometry and are well documented in the 
literature[l8, 24-31]. Absolute reference points in 
space can be established by encoding the grating in 
much the same way that some structured lighting 
techniques employ. 
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Scene analysis technology reasons from the 2-D and 3- 
D information returned by the moire system to 
identify and locate physical objects in the field of 
view [39]. Scene analysis is less complex if the system 
takes advantage of the limited population of potential 
objects to do model-driven recognition. This 
approach requires the system to store a model of each 
object that it may be required to recognize. 
Industrial applications deal with well-defined mass- 
produced parts with consistent characteristics, and so 
lend themselves well to a model-based approach [36]. 

An object can be represented with a semantic net 
formalism modeled on ACRONYM’S prediction graph 
[33]. An object is represented as a set of features , 
such as edges and faces, together with relationships 
between those features. 

Each frame in the net corresponds to a feature , with 
slots for the characteristics of the feature. A frame 
that corresponds to a line records the length of the 
line. A frame that corresponds to a face records such 
information as its area, perimeter, first moment of 
area, centroid, .minimum and maximum diameters, 
and dimensions of smallest enclosing rectangle. In 
some cases, entire components of an object may be 
usefully modeled as a frame recording their extended 
gaussian images [40]. The range and gradient data 
provided by the moire sensor lends itself readily to 
extracting region-based analysis useful for 
characterizing faces of an object In this way. Region- 
growing on the basis of intensity information is 
difficult, since shadows or other secondary 

Interference can produce spurious differences. Data 
showing constant gradient and continuous range is a 
much more robust indicator of a continuous plane 
face. Furthermore, the range and gradient data 
allows us to normalize the slot values recorded for a 
face so that they are invariant with respect to 
rotation or tilting of the part, within reasonable 
limits. Thus the use of a range and gradient image 
permits the construction of a feature-based 
representation of the object that is much more 
detailed and discriminatory than would be feasible 
from monocular intensity data alone. 

Often the characteristics of a single feature will not be 
enough to diagnose the presence of a part, and the 
relationships among features must be taken into 
account. These relationships are represented as arcs 
in the semantic net, and include 

• adjacency between features (lines to lines; 
faces to faces; lines bounding faces); 

• the angles between adjacent lines or 
between the gradients of adjacent faces; 

• mutual exclusion between features that 
cannot be simultaneously visible (such as 
two ends of a brick); 

• Euclidean distance in 3-space between the 
centroids of two features. 

The range and gradient data from the moire sensor 


permits efficient computation of the angular and 
normalized distance relations between features, data 
that otherwise would be very inefficient to obtain. 

The network has nodes not only for features, but also 
for objects . Each feature that belongs to an object is 
joined to it with a M part-of" arc. The object node 
provides a place to store information such as gripping 
points for the object, indexed by the features that are 
visible from each orientation. 

Our strategy is iterative, seeking to identify only one 
or a few objects in the scene, removing it, and then 
acquiring and analyzing a new scene. Some scene- 
analysis strategies seek to identify every item in the 
scene from a single image [37, 32]. The combinatorial 
complexity of matching features to models can 
become very high in this case. Furthermore, once a 
robot begins to manipulate parts in the bin, it may 
inadvertently move other parts, so that a repeated 
analysis may well be necessary anyway. Because 
moire gives range data, we can immediately identify 
the topmost features in the field of view and focus our 
attention on the objects to which they belong. Once 
the robot removes these objects, we take a new view 
of the bin and repeat the process. Since moire yields 
a full-field range image with a single exposure and 
very little computation, acquiring repeated images is 
cheap, and the combinatorial simplification obtained 
by identifying only a few objects at a time reduces the 
cost of analyzing multiplo images. 

One disadvantage of an iterative approach Is that the 
robot cannot pick up parts In the order required by a 
sequential assembly plan, but must take them "as 
they come." There are solutions to this problem. 
The robot may buffer the parts for later reordering. 
Furthermore, if the robot is scheduled 
opportunistically rather than sequentially [34, 35], the 
probability increases that it will be able to make use 
of what it finds on top of the bin. 

As a model-driven program, the system recognizes 
objects by comparing features visible in the frame 
with the prediction graphs stored for expected parts. 
The system does not aimlessly analyze every region 
visible in its field, but begins with the highest pixels 
(those nearest the sensor) to define the "topmost" 
features. As these are matched with features from the 
prediction graphs, graphs that are candidates for 
matching suggest additional features with known 
relations to the ones already observed that the system 
searches for to confirm or disprove hypothetical 
identifications. The use of expectations to drive the 
analysis of image data can significantly reduce the 
amount of visual processing necessary to reach an 
identification. 

The system identifies regions with models by 
constraint propagation , along the lines used by [38]. 
Any single region may match several different models. 
The system maintains with each region a list of the 
objects and their features that the region matches. 
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As more and more regions are matched, the system 
compares the relationships between adjacent regions 
with the relationships specified in the prediction 
graphs for the candidate objects, and refines its 
estimates of the objects to which the features belong 
on the basis of those relationships. 

For example, assume that two regions are adjacent, 
and both match features from two objects, A and 
B. If the relationship between the regions matches the 
relationship between the corresponding features in A 
but not in B, the system refines the identification of 
the regions by eliminating B from the list of candidate 
objects. 


APPLICATIONS 

The distinct characteristics of moire machine vision 
make it particularly suitable to many industrial 
applications where other systems would have 
problems. Unlike systems which depend on surface 
marks or shading effects, moire contouring is very 
amenable to dirty or hostile environments. Such 
environments often exist in automated manufacturing. 
Moire contouring could also be applied in situations 
where the part may be very hot or vibrating where 
neither physical contact nor long scanning 
measurement times are practical. An example of this 
would be a metal extruding operation or an operation 
in which the parts have been freshly painted or 
lacquered. A moire vision system would permit such 
operations to be better controlled by providing 
inspection during otherwise inactive waiting times in 
the manufacturing process. This would reduce waste 
and increase productivity, while removing human 
inspectors from a potentially hazardous environment. 

The noncontact measurement capability of the moire 
machine vision system would have great value in 
many precision measurement applications. As a 
specific example of the need for a full-field optical 
measurement system, NASA has encountered a 
problem in measuring aircraft models for wind tunnel 
tests. The complexity of these models requires that 
they be completely measured to an accuracy of about 
0.001 inches. To measure the models point by point 
manually requires two to three days at considerable 
expense. If any small anomalies are missed, the model 
may not react as predicted in the wind tunnel or may 
fail completely. In addition, because of the delicate 
nature of the surfaces of the models, physical contact 
may actually damage the model. The moire vision 
system could measure the entire model in minutes and 
could even be used as a diagnostic tool during a wind 
tunnel test to measure the model deflections (a task 
impossible for a coordinate measuring machine). 
NASA has been very interested in such technology. 

Another specific example of great current interest in 
the aerospace industry is the inspection of turbine and 
compressor blades for turbine engines. In this case, 
there are a number of different inspection 

requirements. For example, on the leading edge and 


on a region from 0.03 to 0.1 inches back from the 
leading edge center the tolerance is typically on the 
order of 0.0005 inches. On the concave and convex 
faces of the airfoils, which typically have a cord width 
of 2 inches or less, the tolerances are typically in the 
range of 0.002 inches (for some finished turbine 
blades) to 0.004 inches. These inspections are often 
performed using mechanical reference slides by 
measuring the gap between the airfoil and the slide 
with feeler gages. This process is laborious and highly 
dependent on the accuracy of the reference slide, the 
feeler gage and positioning. This is typically purely a 
Go/No-Go type of test. 

With the capability to easily vary the sensitivity of 
moire interferometry, and the high sensitivity of 
moire methods over small areas, a system could be 
developed which would zoom in on the leading edge 
area with high sensitivity and yet be versatile enough 
to provide only the 0.002 inch tolerance measurement 
needed over areas of 2 inches square or more. 

Finally, since the moire data is full-field, and 
available (with the encoding) in one video frame, the 
data can actually be taken in microseconds with a 
strobed camera or lighting. This allows freezing of 
dynamic objects. As the data is built up with 
scanning in the standard line of light structured light 
sensors, the time to obtain the data is 0.1 to 0.3 
seconds, thus precluding the freezing of dynamic 
objects. 

In the area of robot guidance, a moire assisted vision 
system offers a distinct advantage in both speed and 
size over current three-dimensional vision systems or 
range finder gages. The optical preprocessing would 
make real-time three-dimensional information 
available for locating identified parts in a pile (which 
part is on top of the other) or for distinguishing parts 
with the same two-dimensional shape but different 
thickness. Since the third dimension is obtained 
through the vision system, the sensor can be a 
compact, light-weight, solid-state camera such as has 
already been applied to robot guidance and have even 
been mounted directly on the robot arm in some 
applications. 


SUMMARY 

Moire contouring can provide high resolution depth 
information to a vision system by encoding the 
information into an easily analyzed 2-D pattern. The 
depth and surface feature information can be easily 
separated for simplified analysis. Absolute contouring 
information regarding the sign of a slope and the 
relative position of the subject to some reference 
surface is available with this technique. Full-field 
data measuring all points on the object, not just 
where there is a reference point, with variable 
sensitivity and insensitivity to dirt, stray light or 
vibrations is available with moire contouring. A 
moire vision system would solve many of the problems 
now encountered by 3-D vision systems using 
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technology which has been well developed for other 
applications. The application of scene analysis 
techniques similar to those used for 2-D images, can 
provide a means of part recognition, and three- 
dimensional locating tasks required for true flexibility 
of robotic operations. 
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